Skip to content

{tools}[foss/2025b] PyTorch v2.9.1, parameterized v0.9.0, pytest-subtests v0.15.0, ... w/ CUDA 12.9.1#24926

Open
Flamefire wants to merge 35 commits intoeasybuilders:developfrom
Flamefire:20251218180340_new_pr_parameterized090
Open

{tools}[foss/2025b] PyTorch v2.9.1, parameterized v0.9.0, pytest-subtests v0.15.0, ... w/ CUDA 12.9.1#24926
Flamefire wants to merge 35 commits intoeasybuilders:developfrom
Flamefire:20251218180340_new_pr_parameterized090

Conversation

@Flamefire
Copy link
Copy Markdown
Contributor

@Flamefire Flamefire commented Dec 18, 2025

(created using eb --new-pr)

…tests-0.15.0-GCCcore-14.3.0.eb, PyTorch-2.9.1-foss-2025b-CUDA-12.9.1.eb, unittest-xml-reporting-3.2.0-GCCcore-14.3.0.eb and patches: PyTorch-1.12.1_add-hypothesis-suppression.patch, PyTorch-1.7.0_disable-dev-shm-test.patch, PyTorch-2.0.1_skip-tests-skipped-in-subprocess.patch, PyTorch-2.1.0_remove-test-requiring-online-access.patch, PyTorch-2.6.0_show-test-duration.patch, PyTorch-2.6.0_skip-test_segfault.patch, PyTorch-2.7.0_avoid_caffe2_test_cpp_jit.patch, PyTorch-2.7.1_avoid-caffe2-sandcastle-test-lib.patch, PyTorch-2.7.1_skip-test_data_parallel_rnn.patch, PyTorch-2.7.1_skip-test_gds_fails_in_ci.patch, PyTorch-2.7.1_skip-test_mixed_mm_exhaustive_dtypes.patch, PyTorch-2.7.1_skip-tests-requiring-SM90.patch, PyTorch-2.7.1_suport-64bit-BARs.patch, PyTorch-2.7.1_tolerance-test_partial_flat_weights.patch, PyTorch-2.9.0_disable-test_nan_assert.patch, PyTorch-2.9.0_enable-symbolizer-in-test_workspace_allocation_error.patch, PyTorch-2.9.0_fix-attention-squeeze.patch, PyTorch-2.9.0_fix-FP16-CPU-tests-in-test_torchinductor_opinfo.patch, PyTorch-2.9.0_fix-nccl-test-env.patch, PyTorch-2.9.0_fix-test_exclude_padding.patch, PyTorch-2.9.0_fix-test_version_error.patch, PyTorch-2.9.0_honor-XDG_CACHE_HOME.patch, PyTorch-2.9.0_increase-tolerance-in-test_transformers.patch, PyTorch-2.9.0_remove-faulty-close.patch, PyTorch-2.9.0_revert-pybind11-3-change.patch, PyTorch-2.9.0_skip-test_benchmark_on_non_zero_device.patch, PyTorch-2.9.0_skip-test_convolution1-on-H100.patch, PyTorch-2.9.0_skip-test_inductor_all_gather_into_tensor_coalesced.patch, PyTorch-2.9.0_skip-test_original_aten_preserved_pad_mm.patch, PyTorch-2.9.0_skip-test_override-without-CUDA.patch, PyTorch-2.9.0_skip-test_unbacked_reduction.patch, PyTorch-2.9.0_skip-tests-requiring-CUDA-12.8.patch, PyTorch-2.9.0_skip-unexpected-success-in-test_fake_export.patch, PyTorch-2.9.1_skip-RingFlexAttentionTest.patch
@github-actions github-actions Bot added 2025b issues & PRs related to 2025b common toolchains update labels Dec 18, 2025
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Dec 18, 2025

Updated software PyTorch-2.9.1-foss-2025b-CUDA-12.9.1.eb

Diff against PyTorch-2.9.1-foss-2024a.eb

easybuild/easyconfigs/p/PyTorch/PyTorch-2.9.1-foss-2024a.eb

diff --git a/easybuild/easyconfigs/p/PyTorch/PyTorch-2.9.1-foss-2024a.eb b/easybuild/easyconfigs/p/PyTorch/PyTorch-2.9.1-foss-2025b-CUDA-12.9.1.eb
index 83383bd798..e38d4e84ff 100644
--- a/easybuild/easyconfigs/p/PyTorch/PyTorch-2.9.1-foss-2024a.eb
+++ b/easybuild/easyconfigs/p/PyTorch/PyTorch-2.9.1-foss-2025b-CUDA-12.9.1.eb
@@ -1,11 +1,12 @@
 name = 'PyTorch'
 version = '2.9.1'
+versionsuffix = '-CUDA-%(cudaver)s'
 
 homepage = 'https://pytorch.org/'
 description = """Tensors and Dynamic neural networks in Python with strong GPU acceleration.
 PyTorch is a deep learning framework that puts Python first."""
 
-toolchain = {'name': 'foss', 'version': '2024a'}
+toolchain = {'name': 'foss', 'version': '2025b'}
 
 local_six_version = '1.11.0'
 # This is specific to a (tagged) release.
@@ -25,7 +26,7 @@ sources = [
         # Avoid downloading this during the build, see third_party/NNPACK/cmake/DownloadSix.cmake for the version
         'filename': f'six-{local_six_version}.tar.gz',
         'source_urls': ['https://pypi.python.org/packages/source/s/six'],
-    }
+    },
 ]
 patches = [
     'PyTorch-1.12.1_add-hypothesis-suppression.patch',
@@ -56,32 +57,37 @@ patches = [
     'PyTorch-2.9.0_revert-pybind11-3-change.patch',
     'PyTorch-2.9.0_skip-test_benchmark_on_non_zero_device.patch',
     'PyTorch-2.9.0_skip-test_convolution1-on-H100.patch',
+    'PyTorch-2.9.1_skip-test_dtensor_op_db_nn_functional_multi_head_attention_forward_cpu_float32.patch',
     'PyTorch-2.9.0_skip-test_inductor_all_gather_into_tensor_coalesced.patch',
     'PyTorch-2.9.0_skip-test_original_aten_preserved_pad_mm.patch',
     'PyTorch-2.9.0_skip-test_override-without-CUDA.patch',
-    'PyTorch-2.9.0_skip-tests-requiring-CUDA-12.8.patch',
     'PyTorch-2.9.0_skip-test_unbacked_reduction.patch',
+    'PyTorch-2.9.0_skip-tests-requiring-CUDA-12.8.patch',
     'PyTorch-2.9.0_skip-unexpected-success-in-test_fake_export.patch',
-    'PyTorch-2.9.0_update-exptected-output-for-z3-4.13.0.patch',
     'PyTorch-2.9.1_avoid-multiprocess-tests-hanging-forever.patch',
     'PyTorch-2.9.1_avoid-using-wrong-libomp.patch',
     'PyTorch-2.9.1_check-device-avail-test_schedule.patch',
     'PyTorch-2.9.1_disable-slow-tests.patch',
     'PyTorch-2.9.1_dont-print-test-items.patch',
+    'PyTorch-2.9.1_fix-DDPCommHookType-python-3.13.patch',
     'PyTorch-2.9.1_fix-hypothesis-deadline.patch',
     'PyTorch-2.9.1_fix-iteration-in-fligh-reporter.patch',
-    'PyTorch-2.9.1_fix-test_dist2-decorators.patch',
+    'PyTorch-2.9.1_fix-pickle-error-on-Python-3.13.patch',
     'PyTorch-2.9.1_fix-TestExportOpInfoCPU-with-single-GPU.patch',
+    'PyTorch-2.9.1_fix-test_dist2-decorators.patch',
+    'PyTorch-2.9.1_fix-test_recursion_in_except_handler.patch',
     'PyTorch-2.9.1_GCC14-ARM-workaround.patch',
     'PyTorch-2.9.1_ignore-warning-incompatible-pointer-types.patch',
     'PyTorch-2.9.1_normalize_tree_output.patch',
     'PyTorch-2.9.1_set-test-timeout.patch',
+    'PyTorch-2.9.1_skip-bool-bessel-tests.patch',
     'PyTorch-2.9.1_skip-cutlass-addmm-test.patch',
-    'PyTorch-2.9.1_skip-flex-attention-test_block_mask_non_divisible.patch',
     'PyTorch-2.9.1_skip-flex-attention-tests-on-unsupported-cpus.patch',
-    'PyTorch-2.9.1_skip-RingFlexAttentionTest.patch',
-    'PyTorch-2.9.1_skip-test_dtensor_op_db_nn_functional_multi_head_attention_forward_cpu_float32.patch',
+    'PyTorch-2.9.1_skip-flex-attention-test_block_mask_non_divisible.patch',
+    'PyTorch-2.9.1_skip-test_checkpoint_save_failure_continues_serving.patch',
+    'PyTorch-2.9.1_skip-test_norm_matrix_degenerate_shapes.patch',
     'PyTorch-2.9.1_skip-tests-requiring-SM90.patch',
+    'PyTorch-2.9.1_skip-RingFlexAttentionTest.patch',
     'PyTorch-2.9.1_increase-tolerance-TestDecomp-matmul.patch',
     'PyTorch-2.9.1_skip-cpu_repro-tests-failing-on-ARM.patch',
     'PyTorch-2.9.1_skip-svd-pca-lowrank-tests-on-cpu.patch',
@@ -138,20 +144,20 @@ checksums = [
      '85e236431d1a5da3fb7fccc2554640898c29f5fab46a41d15b3ab61dd1f924fc'},
     {'PyTorch-2.9.0_skip-test_convolution1-on-H100.patch':
      '704750c7cc08b58779907d608cd4b7505043e394fb27530b16d72a0dc27c277e'},
+    {'PyTorch-2.9.1_skip-test_dtensor_op_db_nn_functional_multi_head_attention_forward_cpu_float32.patch':
+     'e57486cc42f3dbcae29753168febc251d070a283229e2d76ccbdf19fee53f06e'},
     {'PyTorch-2.9.0_skip-test_inductor_all_gather_into_tensor_coalesced.patch':
      '644153d4c1d8267c0631df2902a6dfe8ec2a197f3374f2a2f5654e6bd0edc05e'},
     {'PyTorch-2.9.0_skip-test_original_aten_preserved_pad_mm.patch':
      'ac9e05d296cd5ff938a44662cd022efcc8133c744ca82b045c6a15bc64f67cf4'},
     {'PyTorch-2.9.0_skip-test_override-without-CUDA.patch':
      '967512d1487bf1ad06982cc5b976c0b38ba062c3f3473cb4542c4b9ac0740662'},
-    {'PyTorch-2.9.0_skip-tests-requiring-CUDA-12.8.patch':
-     '6d79aff5291627b86d8fea025bf2379e4065c7d9cbef5cf83452c35922848728'},
     {'PyTorch-2.9.0_skip-test_unbacked_reduction.patch':
      'b51dd5d7c9cfeed946cbc5c7fc22f2e78e1fa52dda55569b957c20ca4ed01fe8'},
+    {'PyTorch-2.9.0_skip-tests-requiring-CUDA-12.8.patch':
+     '6d79aff5291627b86d8fea025bf2379e4065c7d9cbef5cf83452c35922848728'},
     {'PyTorch-2.9.0_skip-unexpected-success-in-test_fake_export.patch':
      '2e73f71ea0f09e613cc4a108893e7948b6daf239e3fe42fd2d3ae5d43c3cf9de'},
-    {'PyTorch-2.9.0_update-exptected-output-for-z3-4.13.0.patch':
-     '5c68e0de73212ed266879f4528a6041ef7ab2f1ac83c6cf7142c4baa78e7664c'},
     {'PyTorch-2.9.1_avoid-multiprocess-tests-hanging-forever.patch':
      '86ce380e69b3b20e010d817889cb1b825b05b4054a045b00f2ac12161b77d7e4'},
     {'PyTorch-2.9.1_avoid-using-wrong-libomp.patch':
@@ -160,30 +166,40 @@ checksums = [
      '64c28d38ce69147565509add36d310473ce46f14a0a876d38b5049cb7fce9817'},
     {'PyTorch-2.9.1_disable-slow-tests.patch': '6b365a3531b0ac5446b5f0e8ab924b5e5742cd0331e6d9ec979118a3ef0ffc09'},
     {'PyTorch-2.9.1_dont-print-test-items.patch': '2b524cf3d557c0672feefc3a7165e5555e549b0720647a84d546f769cea0be07'},
-    {'PyTorch-2.9.1_fix-hypothesis-deadline.patch': 'f7a130669eee9924a303df9e2bd5743ff023a7d994b7a3e43c86dcccf0206c49'},
+    {'PyTorch-2.9.1_fix-DDPCommHookType-python-3.13.patch':
+     'd7bafe8340bba9dd909475fc62b739b0ce3f95d3409479ef8c5929351dd2a05d'},
+    {'PyTorch-2.9.1_fix-hypothesis-deadline.patch':
+     'f7a130669eee9924a303df9e2bd5743ff023a7d994b7a3e43c86dcccf0206c49'},
     {'PyTorch-2.9.1_fix-iteration-in-fligh-reporter.patch':
      'ab408275ec66e836112a50054acc4e789ef38196efeb6137c6061d60d9ac9ead'},
-    {'PyTorch-2.9.1_fix-test_dist2-decorators.patch':
-     'bf4ed805f00775ed33351de7bce40ebf4eac16aff6c61d2e91790982bc43d73b'},
+    {'PyTorch-2.9.1_fix-pickle-error-on-Python-3.13.patch':
+     '88807b5564485968de3be6411d33c257c5ce59f5d3db23c7aeba884458102d57'},
     {'PyTorch-2.9.1_fix-TestExportOpInfoCPU-with-single-GPU.patch':
      'bdddf5a9ba47d57ec96f4bbefc3b85c4904e44de93dc5c7a65bc03e343035ae9'},
+    {'PyTorch-2.9.1_fix-test_dist2-decorators.patch':
+     'bf4ed805f00775ed33351de7bce40ebf4eac16aff6c61d2e91790982bc43d73b'},
+    {'PyTorch-2.9.1_fix-test_recursion_in_except_handler.patch':
+     'e7a64dbdc202151c5bff6aac86d77b0f6e7c52dc3117e3bfe9b57ec1371f87ad'},
     {'PyTorch-2.9.1_GCC14-ARM-workaround.patch': 'ea8a8662e20fae2fb3a74c7f8bf390aba80a598ab37f9131c720d25ebb14965d'},
     {'PyTorch-2.9.1_ignore-warning-incompatible-pointer-types.patch':
      'c4dad43a5d76e292bb0cada56ea05e8cbd522e3e83749cf3b2c15cd1e4ff6d7b'},
     {'PyTorch-2.9.1_normalize_tree_output.patch': '7d5994580339b73c28de595d9e5a0448db97b7d284f17efd18909e4613d170df'},
     {'PyTorch-2.9.1_set-test-timeout.patch': '15fa1149c250b1333b0bc491f659aaf89d5d6eaf6df5ebc81eea545478c1239c'},
+    {'PyTorch-2.9.1_skip-bool-bessel-tests.patch': '9c07cddaf4c1b17fe9a54874f10b8edbfb85785c40ac1e3aea11c7f4b5abca69'},
     {'PyTorch-2.9.1_skip-cutlass-addmm-test.patch':
      '1f81a8a9eea8eda51fc93dff84cd994772febf4fd05d77efbf21b8440dadfd4e'},
-    {'PyTorch-2.9.1_skip-flex-attention-test_block_mask_non_divisible.patch':
-     'd8489c192da549083569e09e5f94d2a83c9e41e111b1322f86512a9c5a58c0d9'},
     {'PyTorch-2.9.1_skip-flex-attention-tests-on-unsupported-cpus.patch':
      'e544f765beac7bdb3fc0ada98a3f92fd7e511ed8874de085aa2f213cca769d40'},
-    {'PyTorch-2.9.1_skip-RingFlexAttentionTest.patch':
-     '3cf0b11136fb18c45072687eafd3024d91b504d231a4fa40e04bc62d8d6019c7'},
-    {'PyTorch-2.9.1_skip-test_dtensor_op_db_nn_functional_multi_head_attention_forward_cpu_float32.patch':
-     'e57486cc42f3dbcae29753168febc251d070a283229e2d76ccbdf19fee53f06e'},
+    {'PyTorch-2.9.1_skip-flex-attention-test_block_mask_non_divisible.patch':
+     'd8489c192da549083569e09e5f94d2a83c9e41e111b1322f86512a9c5a58c0d9'},
+    {'PyTorch-2.9.1_skip-test_checkpoint_save_failure_continues_serving.patch':
+     'fa22d7ed5bf20afa4798c8af3ec732b1a3f530ecc4be5c223b3796e839b0b812'},
+    {'PyTorch-2.9.1_skip-test_norm_matrix_degenerate_shapes.patch':
+     'd6082e62696a38dbfbc87c228f7ccb54dba4cfc615ce158f1f3bf77e6e30ff4f'},
     {'PyTorch-2.9.1_skip-tests-requiring-SM90.patch':
      '7db02152db2ae70c0fd4c4602fe381e26a74b8e4f7b16b1a3554b2353d761b10'},
+    {'PyTorch-2.9.1_skip-RingFlexAttentionTest.patch':
+     '3cf0b11136fb18c45072687eafd3024d91b504d231a4fa40e04bc62d8d6019c7'},
     {'PyTorch-2.9.1_increase-tolerance-TestDecomp-matmul.patch':
      'dd82203ce3b6262255aba6b59fb3b547c4c17875d5711f6d3d489aa8f0f59f32'},
     {'PyTorch-2.9.1_skip-cpu_repro-tests-failing-on-ARM.patch':
@@ -199,38 +215,47 @@ checksums = [
 osdependencies = [OS_PKG_IBVERBS_DEV]
 
 builddependencies = [
-    ('CMake', '3.29.3'),
-    ('hypothesis', '6.103.1'),
-    ('setuptools', '80.9.0'),
+    ('CMake', '3.31.8'),
+    ('hypothesis', '6.136.6'),
     # For tests
     ('parameterized', '0.9.0'),
     ('pytest-flakefinder', '1.1.0'),
-    ('pytest-rerunfailures', '15.0'),
-    ('pytest-subtests', '0.13.1'),
-    ('tlparse', '0.4.0'),
-    ('optree', '0.14.1'),
-    ('unittest-xml-reporting', '3.1.0'),
+    ('pytest-rerunfailures', '16.1'),
+    ('pytest-subtests', '0.15.0'),
+    ('tlparse', '0.4.3'),
+    ('optree', '0.18.0'),
+    ('unittest-xml-reporting', '3.2.0'),
 ]
 
 dependencies = [
-    ('Ninja', '1.12.1'),  # Required for JIT compilation of C++ extensions
-    ('Python', '3.12.3'),
-    ('Python-bundle-PyPI', '2024.06'),
-    ('expecttest', '0.2.1'),
+    ('CUDA', '12.9.1', '', SYSTEM),
+    # PyTorch is very sensitive to the NCCL & cuDNN versions. (Maybe the same for cuSPARSELt)
+    # Prefer those (listed per CUDA version) in
+    # https://github.com/pytorch/pytorch/blob/main/.github/scripts/generate_binary_build_matrix.py
+    # or https://github.com/pytorch/pytorch/blob/main/.ci/docker/common/install_cuda.sh
+    ('NCCL', '2.27.7', '-CUDA-%(cudaver)s'),
+    ('cuDNN', '9.15.0.57', '-CUDA-%(cudaver)s', SYSTEM),
+    ('magma', '2.9.0', '-CUDA-%(cudaver)s'),
+    ('cuSPARSELt', '0.8.0.4', '-CUDA-%(cudaver)s', SYSTEM),
+    ('Triton', '3.5.0', '-CUDA-%(cudaver)s'),
+    ('Ninja', '1.13.0'),
+    ('Python', '3.13.5'),
+    ('Python-bundle-PyPI', '2025.07'),
+    ('expecttest', '0.3.0'),
     ('GMP', '6.3.0'),
-    ('MPFR', '4.2.1'),
-    ('networkx', '3.4.2'),
-    ('numactl', '2.0.18'),
-    ('Pillow', '10.4.0'),
-    ('protobuf-python', '5.28.0'),
-    ('protobuf', '28.0'),
-    ('pybind11', '2.12.0'),
-    ('PuLP', '2.8.0'),
+    ('MPFR', '4.2.2'),
+    ('networkx', '3.5'),
+    ('numactl', '2.0.19'),
+    ('Pillow', '11.3.0'),
+    ('protobuf-python', '6.31.1'),
+    ('protobuf', '31.1'),
+    ('pybind11', '3.0.0'),
+    ('PuLP', '3.3.0'),
     ('PyYAML', '6.0.2'),
-    ('pyzstd', '0.16.2'),
-    ('SciPy-bundle', '2024.05'),
-    ('sympy', '1.13.3'),
-    ('Z3', '4.13.0',),
+    ('pyzstd', '0.19.0'),
+    ('SciPy-bundle', '2025.07'),
+    ('sympy', '1.14.0'),
+    ('Z3', '4.15.1'),
 ]
 
 prebuildopts = (f"""sed -i '1i set(PYTHON_SIX_SOURCE_DIR "%(builddir)s/six-{local_six_version}")' """
@@ -263,8 +288,6 @@ excluded_tests = {
         'dynamo/test_utils',
         # Packaging test only, not important for us
         'test_license',
-        # No triton
-        'distributed/test_nvshmem_triton',
         # Occasional segfaults on CPU
         'inductor/test_flex_attention',
         'inductor/test_flex_decoding ',
@@ -272,6 +295,7 @@ excluded_tests = {
 }
 
 runtest = (
+    # Disable symbol resolution in stack traces that can cause hangs and slowdowns
     ' TORCH_DISABLE_ADDR2LINE=1'
     ' TORCHINDUCTOR_CUTLASS_DIR=%(start_dir)s/third_party/cutlass'
     ' PYTEST_ADDOPTS=--full-trace'
@@ -290,4 +314,6 @@ modextrapaths = {'TORCHINDUCTOR_CUTLASS_DIR': 'extra/cutlass'}
 
 tests = ['PyTorch-check-cpp-extension.py', 'PyTorch-check-cutlass.py']
 
+sanity_check_pip_list = False
+
 moduleclass = 'ai'
Diff against PyTorch-2.7.1-foss-2024a-CUDA-12.6.0.eb

easybuild/easyconfigs/p/PyTorch/PyTorch-2.7.1-foss-2024a-CUDA-12.6.0.eb

diff --git a/easybuild/easyconfigs/p/PyTorch/PyTorch-2.7.1-foss-2024a-CUDA-12.6.0.eb b/easybuild/easyconfigs/p/PyTorch/PyTorch-2.9.1-foss-2025b-CUDA-12.9.1.eb
index 8ae5323874..e38d4e84ff 100644
--- a/easybuild/easyconfigs/p/PyTorch/PyTorch-2.7.1-foss-2024a-CUDA-12.6.0.eb
+++ b/easybuild/easyconfigs/p/PyTorch/PyTorch-2.9.1-foss-2025b-CUDA-12.9.1.eb
@@ -1,261 +1,265 @@
 name = 'PyTorch'
-version = '2.7.1'
+version = '2.9.1'
 versionsuffix = '-CUDA-%(cudaver)s'
 
 homepage = 'https://pytorch.org/'
 description = """Tensors and Dynamic neural networks in Python with strong GPU acceleration.
 PyTorch is a deep learning framework that puts Python first."""
 
-toolchain = {'name': 'foss', 'version': '2024a'}
+toolchain = {'name': 'foss', 'version': '2025b'}
 
 local_six_version = '1.11.0'
+# This is specific to a (tagged) release.
+# Extract from `get_disabled_tests` in tools/stats/import_test_stats.py
+local_disabled_tests_S3_ID = 'UsscdNP.2GMOzUxAvqIx8GAj4MuhX1Xi'
 source_urls = [GITHUB_RELEASE]
 sources = [
     '%(namelower)s-v%(version)s.tar.gz',
+    {
+        'filename': '%(name)s-%(version)s-disabled-tests.json',
+        'download_filename': f'disabled-tests-condensed.json?versionId={local_disabled_tests_S3_ID}',
+        'source_urls': ['https://ossci-metrics.s3.amazonaws.com'],
+        # See `DEFAULT_DISABLED_TESTS_FILE` in torch/testing/_internal/common_utils.py
+        'extract_cmd': 'cp %s %(builddir)s/pytorch-v%(version)s/test/.pytorch-disabled-tests.json',
+    },
     {
         # Avoid downloading this during the build, see third_party/NNPACK/cmake/DownloadSix.cmake for the version
         'filename': f'six-{local_six_version}.tar.gz',
-        'source_urls':
-        ['https://pypi.python.org/packages/16/d8/bc6316cf98419719bd59c91742194c111b6f2e85abac88e496adefaf7afe'],
-    }
+        'source_urls': ['https://pypi.python.org/packages/source/s/six'],
+    },
 ]
 patches = [
     'PyTorch-1.12.1_add-hypothesis-suppression.patch',
-    'PyTorch-1.13.1_skip-failing-singular-grad-test.patch',
     'PyTorch-1.7.0_disable-dev-shm-test.patch',
-    'PyTorch-2.0.1_avoid-test_quantization-failures.patch',
-    'PyTorch-2.0.1_skip-test_shuffle_reproducibility.patch',
     'PyTorch-2.0.1_skip-tests-skipped-in-subprocess.patch',
     'PyTorch-2.1.0_remove-test-requiring-online-access.patch',
-    'PyTorch-2.1.0_skip-dynamo-test_predispatch.patch',
-    'PyTorch-2.1.2_workaround_dynamo_failure_without_nnpack.patch',
-    'PyTorch-2.3.0_disable_test_linear_package_if_no_half_types_are_available.patch',
-    'PyTorch-2.3.0_skip_test_var_mean_differentiable.patch',
-    'PyTorch-2.6.0_disable_DataType_dependent_test_if_tensorboard_is_not_available.patch',
-    'PyTorch-2.6.0_disable_tests_which_need_network_download.patch',
-    'PyTorch-2.6.0_disable-gcc12-warnings.patch',
-    'PyTorch-2.6.0_fix-accuracy-issues-in-linalg_solve.patch',
     'PyTorch-2.6.0_fix-server-in-test_control_plane.patch',
-    'PyTorch-2.6.0_fix-vsx-vector-shift-functions.patch',
-    'PyTorch-2.6.0_increase-tolerance-test_aotdispatch-matmul.patch',
-    'PyTorch-2.6.0_increase-tolerance-test_quick-baddbmm.patch',
-    'PyTorch-2.6.0_increase-tolerance-test_vmap_autograd_grad.patch',
     'PyTorch-2.6.0_show-test-duration.patch',
-    'PyTorch-2.6.0_skip-diff-test-on-ppc.patch',
     'PyTorch-2.6.0_skip-test_segfault.patch',
-    'PyTorch-2.6.0_skip-test-requiring-MKL.patch',
     'PyTorch-2.7.0_avoid_caffe2_test_cpp_jit.patch',
-    'PyTorch-2.7.0_do-not-checkout-nccl.patch',
-    'PyTorch-2.7.0_fix-distributed-tests-without-gpus.patch',
-    'PyTorch-2.7.0_fix-skip-decorators.patch',
-    'PyTorch-2.7.0_fix-test_linear_with_embedding.patch',
-    'PyTorch-2.7.0_fix-test_linear_with_in_out_buffer-without-mkl.patch',
-    'PyTorch-2.7.0_skip-test_init_from_local_shards.patch',
     'PyTorch-2.7.1_avoid-caffe2-sandcastle-test-lib.patch',
-    'PyTorch-2.7.1_avoid-using-wrong-libomp.patch',
-    'PyTorch-2.7.1_do-not-rpath-stubs-dir-in-jit-code.patch',
-    'PyTorch-2.7.1_exit-test_c10d_ops_nccl-with-succes-when-no-gpu.patch',
-    'PyTorch-2.7.1_fix-cuda-12.6-driver-api-usage.patch',
-    'PyTorch-2.7.1_fix-CUDASymmetricMemory-group.patch',
-    'PyTorch-2.7.1_fix-nccl-test-env.patch',
-    'PyTorch-2.7.1_fix-test_ck_blas_library_cpu.patch',
-    'PyTorch-2.7.1_fix-test_fsdp_ep.patch',
-    'PyTorch-2.7.1_fix-test_ir_count.patch',
-    'PyTorch-2.7.1_fix-test_torchinductor_dynamic-tests.patch',
-    'PyTorch-2.7.1_increase-tolerance-for-sum-reduction-test.patch',
-    'PyTorch-2.7.1_increase-tolerance-for-test_freeze_conv_relu_fusion.patch',
-    'PyTorch-2.7.1_init-cutlass-include-dirs.patch',
-    'PyTorch-2.7.1_keep-CMAKE_PREFIX_PATH-in-test.patch',
-    'PyTorch-2.7.1_remove-faulty-close.patch',
-    'PyTorch-2.7.1_remove-test_close_multi_pg_unordered.patch',
-    'PyTorch-2.7.1_serialize-test_host_memory_stats.patch',
-    'PyTorch-2.7.1_skip-failing-max_autotune-tests.patch',
-    'PyTorch-2.7.1_skip-failing-schedule-test.patch',
-    'PyTorch-2.7.1_skip-NCCL-tests-without-GPUs.patch',
-    'PyTorch-2.7.1_skip-test_benchmark_on_non_zero_device.patch',
     'PyTorch-2.7.1_skip-test_data_parallel_rnn.patch',
     'PyTorch-2.7.1_skip-test_gds_fails_in_ci.patch',
-    'PyTorch-2.7.1_skip-test_lowering_one_shot_all_reduce.patch',
     'PyTorch-2.7.1_skip-test_mixed_mm_exhaustive_dtypes.patch',
-    'PyTorch-2.7.1_skip-test_non_blocking_with_eager_init.patch',
     'PyTorch-2.7.1_skip-test_outside_linear_module_free.patch',
-    'PyTorch-2.7.1_skip-test_override-without-CUDA.patch',
-    'PyTorch-2.7.1_skip-TestFP8Lowering.patch',
-    'PyTorch-2.7.1_skip-tests-requiring-cc89.patch',
-    'PyTorch-2.7.1_skip-tests-requiring-SM90.patch',
     'PyTorch-2.7.1_suport-64bit-BARs.patch',
     'PyTorch-2.7.1_tolerance-test_partial_flat_weights.patch',
+    'PyTorch-2.9.0_disable-test_nan_assert.patch',
+    'PyTorch-2.9.0_enable-symbolizer-in-test_workspace_allocation_error.patch',
+    'PyTorch-2.9.0_fix-attention-squeeze.patch',
+    'PyTorch-2.9.0_fix-FP16-CPU-tests-in-test_torchinductor_opinfo.patch',
+    'PyTorch-2.9.0_fix-nccl-test-env.patch',
+    'PyTorch-2.9.0_fix-test_exclude_padding.patch',
+    'PyTorch-2.9.0_fix-test_version_error.patch',
+    'PyTorch-2.9.0_honor-XDG_CACHE_HOME.patch',
+    'PyTorch-2.9.0_increase-tolerance-in-test_transformers.patch',
+    'PyTorch-2.9.0_remove-faulty-close.patch',
+    'PyTorch-2.9.0_revert-pybind11-3-change.patch',
+    'PyTorch-2.9.0_skip-test_benchmark_on_non_zero_device.patch',
+    'PyTorch-2.9.0_skip-test_convolution1-on-H100.patch',
+    'PyTorch-2.9.1_skip-test_dtensor_op_db_nn_functional_multi_head_attention_forward_cpu_float32.patch',
+    'PyTorch-2.9.0_skip-test_inductor_all_gather_into_tensor_coalesced.patch',
+    'PyTorch-2.9.0_skip-test_original_aten_preserved_pad_mm.patch',
+    'PyTorch-2.9.0_skip-test_override-without-CUDA.patch',
+    'PyTorch-2.9.0_skip-test_unbacked_reduction.patch',
+    'PyTorch-2.9.0_skip-tests-requiring-CUDA-12.8.patch',
+    'PyTorch-2.9.0_skip-unexpected-success-in-test_fake_export.patch',
+    'PyTorch-2.9.1_avoid-multiprocess-tests-hanging-forever.patch',
+    'PyTorch-2.9.1_avoid-using-wrong-libomp.patch',
+    'PyTorch-2.9.1_check-device-avail-test_schedule.patch',
+    'PyTorch-2.9.1_disable-slow-tests.patch',
+    'PyTorch-2.9.1_dont-print-test-items.patch',
+    'PyTorch-2.9.1_fix-DDPCommHookType-python-3.13.patch',
+    'PyTorch-2.9.1_fix-hypothesis-deadline.patch',
+    'PyTorch-2.9.1_fix-iteration-in-fligh-reporter.patch',
+    'PyTorch-2.9.1_fix-pickle-error-on-Python-3.13.patch',
+    'PyTorch-2.9.1_fix-TestExportOpInfoCPU-with-single-GPU.patch',
+    'PyTorch-2.9.1_fix-test_dist2-decorators.patch',
+    'PyTorch-2.9.1_fix-test_recursion_in_except_handler.patch',
+    'PyTorch-2.9.1_GCC14-ARM-workaround.patch',
+    'PyTorch-2.9.1_ignore-warning-incompatible-pointer-types.patch',
+    'PyTorch-2.9.1_normalize_tree_output.patch',
+    'PyTorch-2.9.1_set-test-timeout.patch',
+    'PyTorch-2.9.1_skip-bool-bessel-tests.patch',
+    'PyTorch-2.9.1_skip-cutlass-addmm-test.patch',
+    'PyTorch-2.9.1_skip-flex-attention-tests-on-unsupported-cpus.patch',
+    'PyTorch-2.9.1_skip-flex-attention-test_block_mask_non_divisible.patch',
+    'PyTorch-2.9.1_skip-test_checkpoint_save_failure_continues_serving.patch',
+    'PyTorch-2.9.1_skip-test_norm_matrix_degenerate_shapes.patch',
+    'PyTorch-2.9.1_skip-tests-requiring-SM90.patch',
+    'PyTorch-2.9.1_skip-RingFlexAttentionTest.patch',
+    'PyTorch-2.9.1_increase-tolerance-TestDecomp-matmul.patch',
+    'PyTorch-2.9.1_skip-cpu_repro-tests-failing-on-ARM.patch',
+    'PyTorch-2.9.1_skip-svd-pca-lowrank-tests-on-cpu.patch',
+    'PyTorch-2.9.1_skip-test_optree_graph_break_message.patch',
+    'PyTorch-2.9.1_skip-tests-requiring-MKLDNN.patch',
 ]
 checksums = [
-    {'pytorch-v2.7.1.tar.gz': '5befd2e540fd55ce4782d0ca7610ce5b572d756d7ea38090ef0f3c7c428fb20f'},
-    {f"six-{local_six_version}.tar.gz": "70e8a77beed4562e7f14fe23a786b54f6296e34344c23bc42f07b15018ff98e9"},
+    {'pytorch-v2.9.1.tar.gz': 'e17504700ebc4c87f9b57059df1c4d790b769458c04db144c7a92aea90f2c92b'},
+    {'PyTorch-2.9.1-disabled-tests.json': '471f8aa36e056173d09ffd421ead45539a8d35fec6e61a8a0050d92a5fcd9f04'},
+    {'six-1.11.0.tar.gz': '70e8a77beed4562e7f14fe23a786b54f6296e34344c23bc42f07b15018ff98e9'},
     {'PyTorch-1.12.1_add-hypothesis-suppression.patch':
      'e71ffb94ebe69f580fa70e0de84017058325fdff944866d6bd03463626edc32c'},
-    {'PyTorch-1.13.1_skip-failing-singular-grad-test.patch':
-     '72688a57b2bb617665ad1a1d5e362c5111ae912c10936bb38a089c0204729f48'},
     {'PyTorch-1.7.0_disable-dev-shm-test.patch': '622cb1eaeadc06e13128a862d9946bcc1f1edd3d02b259c56a9aecc4d5406b8a'},
-    {'PyTorch-2.0.1_avoid-test_quantization-failures.patch':
-     '02e3f47e4ed1d7d6077e26f1ae50073dc2b20426269930b505f4aefe5d2f33cd'},
-    {'PyTorch-2.0.1_skip-test_shuffle_reproducibility.patch':
-     '7047862abc1abaff62954da59700f36d4f39fcf83167a638183b1b7f8fec78ae'},
     {'PyTorch-2.0.1_skip-tests-skipped-in-subprocess.patch':
      '166c134573a95230e39b9ea09ece3ad8072f39d370c9a88fb2a1e24f6aaac2b5'},
     {'PyTorch-2.1.0_remove-test-requiring-online-access.patch':
      '35184b8c5a1b10f79e511cc25db3b8a5585a5d58b5d1aa25dd3d250200b14fd7'},
-    {'PyTorch-2.1.0_skip-dynamo-test_predispatch.patch':
-     '6298daf9ddaa8542850eee9ea005f28594ab65b1f87af43d8aeca1579a8c4354'},
-    {'PyTorch-2.1.2_workaround_dynamo_failure_without_nnpack.patch':
-     'fb96eefabf394617bbb3fbd3a7a7c1aa5991b3836edc2e5d2a30e708bfe49ba1'},
-    {'PyTorch-2.3.0_disable_test_linear_package_if_no_half_types_are_available.patch':
-     '23416f2d9d5226695ec3fbea0671e3650c655c19deefd3f0f8ddab5afa50f485'},
-    {'PyTorch-2.3.0_skip_test_var_mean_differentiable.patch':
-     '9703fd0f1fca8916f6d79d83e9a7efe8e3f717362a5fdaa8f5d9da90d0c75018'},
-    {'PyTorch-2.6.0_disable_DataType_dependent_test_if_tensorboard_is_not_available.patch':
-     '74db866787f1e666ed3b35db5204f05a0ba8d989fb23057a72dd07928388dc46'},
-    {'PyTorch-2.6.0_disable_tests_which_need_network_download.patch':
-     'fe76129811e4eb24d0e12c397335a4c7971b0c4e48ce9cdb9169f3ef9de7aac4'},
-    {'PyTorch-2.6.0_disable-gcc12-warnings.patch': '892643650788b743106ebe4e70c68be42a756eba797f0f79e31708d6e008a620'},
-    {'PyTorch-2.6.0_fix-accuracy-issues-in-linalg_solve.patch':
-     'a6b1cfe8f03ad5b17437e04e6a0369a25fcc79eed939ce6912ceca1c0ab0f444'},
     {'PyTorch-2.6.0_fix-server-in-test_control_plane.patch':
      '1337689ff28ecaa8d1d0edf60d322bcdd7846fec040925325d357b19eb6e4342'},
-    {'PyTorch-2.6.0_fix-vsx-vector-shift-functions.patch':
-     '82ce0b48e3b7c3dfd3a2ba915f4675d5c3a6d149646e1e0d6a29eedbbaecc8bd'},
-    {'PyTorch-2.6.0_increase-tolerance-test_aotdispatch-matmul.patch':
-     'c1c6ea41504e4479d258225ecefc7e9c5726934601610904ae555501a11e9109'},
-    {'PyTorch-2.6.0_increase-tolerance-test_quick-baddbmm.patch':
-     '9850facdfb5d98451249570788217ede07466cae9ba52cd03afd3ec803ba33c9'},
-    {'PyTorch-2.6.0_increase-tolerance-test_vmap_autograd_grad.patch':
-     '8d5eb53bb0a1456af333ae646c860033d6dd037bd9152601a200ca5c10ebf3cb'},
     {'PyTorch-2.6.0_show-test-duration.patch': '5508f2f9619204d9f3c356dbd4000a00d58f452ab2d64ae920eb8bc8b5484d75'},
-    {'PyTorch-2.6.0_skip-diff-test-on-ppc.patch': '6f2f87cad1b0ab8c5a0c7b3f7fbc14e4bdfbe61da26a3934ded9dda7fe368c74'},
     {'PyTorch-2.6.0_skip-test_segfault.patch': '26806bd62e6b61b56ebaa52d68ca44c415a28124f684bd2fb373557ada68ef52'},
-    {'PyTorch-2.6.0_skip-test-requiring-MKL.patch': 'f1c9b1c77b09d59317fd52d390e7d948a147325b927ad6373c1fa1d1d6ea1ea8'},
     {'PyTorch-2.7.0_avoid_caffe2_test_cpp_jit.patch':
      '2f3255e067f5c6f0d78b4fbce94784c41bddf3d01bab9673856b0d0bbc4e3fec'},
-    {'PyTorch-2.7.0_do-not-checkout-nccl.patch': 'ad085a15dd36768ad33a934f53dc595da745e01697b44d431f8b70ae9d0eb567'},
-    {'PyTorch-2.7.0_fix-distributed-tests-without-gpus.patch':
-     '99d92db44f856b2fb05c221f201e50c21e57a7f6f35824f8274a380875029f24'},
-    {'PyTorch-2.7.0_fix-skip-decorators.patch': 'a5197594f8b076f9a2d03ae3aa725018d55889b737a12b74d6872b5c1bd1e809'},
-    {'PyTorch-2.7.0_fix-test_linear_with_embedding.patch':
-     '276b100a4a405fae6a9517cec1ca166b6f8097668f08f7e20aacf3cb766f9a2a'},
-    {'PyTorch-2.7.0_fix-test_linear_with_in_out_buffer-without-mkl.patch':
-     '507931ad00afab098ef9df99ac32c28c61c11ca0e0ac2c55570d9b9e7dc8ef38'},
-    {'PyTorch-2.7.0_skip-test_init_from_local_shards.patch':
-     '655e57763c6ddc3d8b52ed67aaf0f59874441a69161d10c14ab8860f9be5c332'},
     {'PyTorch-2.7.1_avoid-caffe2-sandcastle-test-lib.patch':
      'aaf22cb431357dc78e4db895d64febf1c7ee187e8ad27bd13544d011127354d4'},
-    {'PyTorch-2.7.1_avoid-using-wrong-libomp.patch':
-     'c14daa1351548b22303ec1162e97f11f94bab9ff32a035cf0a814f43374b5f8d'},
-    {'PyTorch-2.7.1_do-not-rpath-stubs-dir-in-jit-code.patch':
-     '6ab92ce23618c74a4950a6dc652d8ea1ff03c101c4f93a9186da29e136b17b1a'},
-    {'PyTorch-2.7.1_exit-test_c10d_ops_nccl-with-succes-when-no-gpu.patch':
-     'ad46a9167ceeafe073618588b2ca13cdef431aae732713b5dc545a93eb9cd076'},
-    {'PyTorch-2.7.1_fix-cuda-12.6-driver-api-usage.patch':
-     '51b2e51ff8419f263f0b9f4352fb503f6f48f2950076f9596b299ff2a0121747'},
-    {'PyTorch-2.7.1_fix-CUDASymmetricMemory-group.patch':
-     '9184b48af7b7caa77f038c911c43cd85f0daa6992f1197adb0ad27b80f5fc40a'},
-    {'PyTorch-2.7.1_fix-nccl-test-env.patch': 'ddb052d217c9811aa2c96e71d52149d2e531b9dfb3b14ca4c7d87d33f54d30cd'},
-    {'PyTorch-2.7.1_fix-test_ck_blas_library_cpu.patch':
-     '9df61b4ed2bd7f4a30df463cd2c5d4cb84d57932909b34dfa360d214425a5fee'},
-    {'PyTorch-2.7.1_fix-test_fsdp_ep.patch': '9cd2da8027e440dd3069fcbd5692703dbfbb9fa9046ebcc5092669a10408b6ef'},
-    {'PyTorch-2.7.1_fix-test_ir_count.patch': 'ba3dc48ee356d48ced89e2d6fceb8c8e91caccd0bda600e4ec4c3540cf434cad'},
-    {'PyTorch-2.7.1_fix-test_torchinductor_dynamic-tests.patch':
-     '1343babbe9fb8a2cc8a12481647340e50f63beffbfa1e92ed4e5a6203f857af4'},
-    {'PyTorch-2.7.1_increase-tolerance-for-sum-reduction-test.patch':
-     '1280259d12a4cf9fcfc22f4b92796072f9ee37b9734c77ebdcbf43d42235d15a'},
-    {'PyTorch-2.7.1_increase-tolerance-for-test_freeze_conv_relu_fusion.patch':
-     'e24f7fa6f43c5ea0fab2c2b876644649948aabae0b2d239e845e20dfd607b7e6'},
-    {'PyTorch-2.7.1_init-cutlass-include-dirs.patch':
-     '682a295519c81afb692caba66eb5e64570f938525e7ded803627884c382d509a'},
-    {'PyTorch-2.7.1_keep-CMAKE_PREFIX_PATH-in-test.patch':
-     '516d0a9a7490999f979eb9e53ae1efd6fdea6ed5f94c9dbd659fb1e5d1fd022b'},
-    {'PyTorch-2.7.1_remove-faulty-close.patch': '315fca3c582534f20da62078156c91b38637f1358cd166b4d33ba964c7b07f95'},
-    {'PyTorch-2.7.1_remove-test_close_multi_pg_unordered.patch':
-     '65a6d430ec359b9fee5f389e05b4c4c592db1ddc12fdab550445b52f3f2a7bfe'},
-    {'PyTorch-2.7.1_serialize-test_host_memory_stats.patch':
-     'ed17602b0458c9d954cfe0c0d7373a2beee13f1ee8eccf3d5f8131980e319ef0'},
-    {'PyTorch-2.7.1_skip-failing-max_autotune-tests.patch':
-     '8611605060088b0178834d34621d407c6ba03803d65e433971f458c05adf0c10'},
-    {'PyTorch-2.7.1_skip-failing-schedule-test.patch':
-     '50151c6792d64c1e01533218b70f0ed974b934334b008e6df558ae8a9b999910'},
-    {'PyTorch-2.7.1_skip-NCCL-tests-without-GPUs.patch':
-     '550c6976b9e3305ceb25cf2de5d135ca771c49acccd2d331c724ade8ccaecde2'},
-    {'PyTorch-2.7.1_skip-test_benchmark_on_non_zero_device.patch':
-     '1c35b207b6fcc24fbfcdc7552fb4f0c9b77233f8a9032a7caa2b8f94d33491d0'},
     {'PyTorch-2.7.1_skip-test_data_parallel_rnn.patch':
      'aa85b678e89db4bb41d2c5f4990f0d05959be92e61918291cb5609685b7f1841'},
     {'PyTorch-2.7.1_skip-test_gds_fails_in_ci.patch':
      '503030c3591196510a3c2d95db30b28a0b396adb8b50ff0d221f6bdb1f939935'},
-    {'PyTorch-2.7.1_skip-test_lowering_one_shot_all_reduce.patch':
-     'c5235fab6cac29adfa61238ddfa71bee18c470e7b3b58f18cc585a1dc3fbeb65'},
     {'PyTorch-2.7.1_skip-test_mixed_mm_exhaustive_dtypes.patch':
      '709288abc802c9eb687c15f2677ebaf408d8325a4cb470d23cb72447ee0b8e13'},
-    {'PyTorch-2.7.1_skip-test_non_blocking_with_eager_init.patch':
-     '5c78fab8e1cf01a8e73731fd38342455fcc2e14304505b779eb998dddbbf1dd8'},
     {'PyTorch-2.7.1_skip-test_outside_linear_module_free.patch':
      '4916a256b2b9914e4fdb930681b80df93ea561ddee2fc9978c4973a5650be5e9'},
-    {'PyTorch-2.7.1_skip-test_override-without-CUDA.patch':
-     'a94654b9ba492be1cef0c8f266d1e16e0c5efb35816164f8c2bfdfda2dfa65f5'},
-    {'PyTorch-2.7.1_skip-TestFP8Lowering.patch': 'a1b5d15795d1c776fa7dca9e3eb8c5335d940e6961cb7d9980d1bfe49b847391'},
-    {'PyTorch-2.7.1_skip-tests-requiring-cc89.patch':
-     'ac39e77339196d734837792791baa058732fb2e87180f6007ae5512028b68659'},
-    {'PyTorch-2.7.1_skip-tests-requiring-SM90.patch':
-     '7b5891a96b58d1d404c130233ec5ddbb0ad52afdb9c334bbe4e1f27f6c78ffd8'},
     {'PyTorch-2.7.1_suport-64bit-BARs.patch': '317c3d220aa87426d86e137a6c1a8f910adf9580ca0848371e0f6800c05dbde1'},
     {'PyTorch-2.7.1_tolerance-test_partial_flat_weights.patch':
      'f304440a57e00b8052a5ffbf285adad8d0fdc5a812a659420b59a20deb5a9942'},
+    {'PyTorch-2.9.0_disable-test_nan_assert.patch': '98e9f98ce8fb89ae368739bc039be69040ed446a1c74ee5c2a1ef8ba60986c7d'},
+    {'PyTorch-2.9.0_enable-symbolizer-in-test_workspace_allocation_error.patch':
+     'ba4032b967c0393c916a26fb2b117ba40670ae8e809cb34399a6379b4e523d72'},
+    {'PyTorch-2.9.0_fix-attention-squeeze.patch': '8f040e74780cab391bb4c84f86390a13230e1a309ddf65db9900d9a1c66e1288'},
+    {'PyTorch-2.9.0_fix-FP16-CPU-tests-in-test_torchinductor_opinfo.patch':
+     'b696d7be8c55ff1ccf8731dccf119b8792cd9593eaff457f37e76114e52346d2'},
+    {'PyTorch-2.9.0_fix-nccl-test-env.patch': '9326223c400262788734ec608f6134c5d240f4d5315a8d294179a28f885d6845'},
+    {'PyTorch-2.9.0_fix-test_exclude_padding.patch':
+     '349850874fb75d57a24437d871a4994a773e501632ce66a2adca613380a152dc'},
+    {'PyTorch-2.9.0_fix-test_version_error.patch': 'b10bb10d0a353e4ba7dbef28ca5fef03a8ba552896e1982708aa90ab6f24f34f'},
+    {'PyTorch-2.9.0_honor-XDG_CACHE_HOME.patch': '239631258431174e4aed8947ae6096e003a3213bfbfa112cd0cdebae89469164'},
+    {'PyTorch-2.9.0_increase-tolerance-in-test_transformers.patch':
+     'c27ab34900835c2a15edc26d481343a16433bfa52f635a80cbab252c1320a545'},
+    {'PyTorch-2.9.0_remove-faulty-close.patch': '32ca744d68dcfa669e46ced9d2776af3dcc380dd9c3458ba7c1c432e5c5295b3'},
+    {'PyTorch-2.9.0_revert-pybind11-3-change.patch':
+     '5289894011fefc67482b1e19c9d1c502e94a943fc7a2d5ed5a6a1eaf444570a0'},
+    {'PyTorch-2.9.0_skip-test_benchmark_on_non_zero_device.patch':
+     '85e236431d1a5da3fb7fccc2554640898c29f5fab46a41d15b3ab61dd1f924fc'},
+    {'PyTorch-2.9.0_skip-test_convolution1-on-H100.patch':
+     '704750c7cc08b58779907d608cd4b7505043e394fb27530b16d72a0dc27c277e'},
+    {'PyTorch-2.9.1_skip-test_dtensor_op_db_nn_functional_multi_head_attention_forward_cpu_float32.patch':
+     'e57486cc42f3dbcae29753168febc251d070a283229e2d76ccbdf19fee53f06e'},
+    {'PyTorch-2.9.0_skip-test_inductor_all_gather_into_tensor_coalesced.patch':
+     '644153d4c1d8267c0631df2902a6dfe8ec2a197f3374f2a2f5654e6bd0edc05e'},
+    {'PyTorch-2.9.0_skip-test_original_aten_preserved_pad_mm.patch':
+     'ac9e05d296cd5ff938a44662cd022efcc8133c744ca82b045c6a15bc64f67cf4'},
+    {'PyTorch-2.9.0_skip-test_override-without-CUDA.patch':
+     '967512d1487bf1ad06982cc5b976c0b38ba062c3f3473cb4542c4b9ac0740662'},
+    {'PyTorch-2.9.0_skip-test_unbacked_reduction.patch':
+     'b51dd5d7c9cfeed946cbc5c7fc22f2e78e1fa52dda55569b957c20ca4ed01fe8'},
+    {'PyTorch-2.9.0_skip-tests-requiring-CUDA-12.8.patch':
+     '6d79aff5291627b86d8fea025bf2379e4065c7d9cbef5cf83452c35922848728'},
+    {'PyTorch-2.9.0_skip-unexpected-success-in-test_fake_export.patch':
+     '2e73f71ea0f09e613cc4a108893e7948b6daf239e3fe42fd2d3ae5d43c3cf9de'},
+    {'PyTorch-2.9.1_avoid-multiprocess-tests-hanging-forever.patch':
+     '86ce380e69b3b20e010d817889cb1b825b05b4054a045b00f2ac12161b77d7e4'},
+    {'PyTorch-2.9.1_avoid-using-wrong-libomp.patch':
+     '2fc2bb82cce87ba0ce73718b0502735ecdf360ca6bfac4482396f7f1c51c1866'},
+    {'PyTorch-2.9.1_check-device-avail-test_schedule.patch':
+     '64c28d38ce69147565509add36d310473ce46f14a0a876d38b5049cb7fce9817'},
+    {'PyTorch-2.9.1_disable-slow-tests.patch': '6b365a3531b0ac5446b5f0e8ab924b5e5742cd0331e6d9ec979118a3ef0ffc09'},
+    {'PyTorch-2.9.1_dont-print-test-items.patch': '2b524cf3d557c0672feefc3a7165e5555e549b0720647a84d546f769cea0be07'},
+    {'PyTorch-2.9.1_fix-DDPCommHookType-python-3.13.patch':
+     'd7bafe8340bba9dd909475fc62b739b0ce3f95d3409479ef8c5929351dd2a05d'},
+    {'PyTorch-2.9.1_fix-hypothesis-deadline.patch':
+     'f7a130669eee9924a303df9e2bd5743ff023a7d994b7a3e43c86dcccf0206c49'},
+    {'PyTorch-2.9.1_fix-iteration-in-fligh-reporter.patch':
+     'ab408275ec66e836112a50054acc4e789ef38196efeb6137c6061d60d9ac9ead'},
+    {'PyTorch-2.9.1_fix-pickle-error-on-Python-3.13.patch':
+     '88807b5564485968de3be6411d33c257c5ce59f5d3db23c7aeba884458102d57'},
+    {'PyTorch-2.9.1_fix-TestExportOpInfoCPU-with-single-GPU.patch':
+     'bdddf5a9ba47d57ec96f4bbefc3b85c4904e44de93dc5c7a65bc03e343035ae9'},
+    {'PyTorch-2.9.1_fix-test_dist2-decorators.patch':
+     'bf4ed805f00775ed33351de7bce40ebf4eac16aff6c61d2e91790982bc43d73b'},
+    {'PyTorch-2.9.1_fix-test_recursion_in_except_handler.patch':
+     'e7a64dbdc202151c5bff6aac86d77b0f6e7c52dc3117e3bfe9b57ec1371f87ad'},
+    {'PyTorch-2.9.1_GCC14-ARM-workaround.patch': 'ea8a8662e20fae2fb3a74c7f8bf390aba80a598ab37f9131c720d25ebb14965d'},
+    {'PyTorch-2.9.1_ignore-warning-incompatible-pointer-types.patch':
+     'c4dad43a5d76e292bb0cada56ea05e8cbd522e3e83749cf3b2c15cd1e4ff6d7b'},
+    {'PyTorch-2.9.1_normalize_tree_output.patch': '7d5994580339b73c28de595d9e5a0448db97b7d284f17efd18909e4613d170df'},
+    {'PyTorch-2.9.1_set-test-timeout.patch': '15fa1149c250b1333b0bc491f659aaf89d5d6eaf6df5ebc81eea545478c1239c'},
+    {'PyTorch-2.9.1_skip-bool-bessel-tests.patch': '9c07cddaf4c1b17fe9a54874f10b8edbfb85785c40ac1e3aea11c7f4b5abca69'},
+    {'PyTorch-2.9.1_skip-cutlass-addmm-test.patch':
+     '1f81a8a9eea8eda51fc93dff84cd994772febf4fd05d77efbf21b8440dadfd4e'},
+    {'PyTorch-2.9.1_skip-flex-attention-tests-on-unsupported-cpus.patch':
+     'e544f765beac7bdb3fc0ada98a3f92fd7e511ed8874de085aa2f213cca769d40'},
+    {'PyTorch-2.9.1_skip-flex-attention-test_block_mask_non_divisible.patch':
+     'd8489c192da549083569e09e5f94d2a83c9e41e111b1322f86512a9c5a58c0d9'},
+    {'PyTorch-2.9.1_skip-test_checkpoint_save_failure_continues_serving.patch':
+     'fa22d7ed5bf20afa4798c8af3ec732b1a3f530ecc4be5c223b3796e839b0b812'},
+    {'PyTorch-2.9.1_skip-test_norm_matrix_degenerate_shapes.patch':
+     'd6082e62696a38dbfbc87c228f7ccb54dba4cfc615ce158f1f3bf77e6e30ff4f'},
+    {'PyTorch-2.9.1_skip-tests-requiring-SM90.patch':
+     '7db02152db2ae70c0fd4c4602fe381e26a74b8e4f7b16b1a3554b2353d761b10'},
+    {'PyTorch-2.9.1_skip-RingFlexAttentionTest.patch':
+     '3cf0b11136fb18c45072687eafd3024d91b504d231a4fa40e04bc62d8d6019c7'},
+    {'PyTorch-2.9.1_increase-tolerance-TestDecomp-matmul.patch':
+     'dd82203ce3b6262255aba6b59fb3b547c4c17875d5711f6d3d489aa8f0f59f32'},
+    {'PyTorch-2.9.1_skip-cpu_repro-tests-failing-on-ARM.patch':
+     '99055fde02ca17c1db1cd72f41821387a50901d6cd947161cafa12257b3a1c5a'},
+    {'PyTorch-2.9.1_skip-svd-pca-lowrank-tests-on-cpu.patch':
+     '4fc772293047dc737b99e232b8a8db904aa8e88e3c8b2bcc3602fb723941fb89'},
+    {'PyTorch-2.9.1_skip-test_optree_graph_break_message.patch':
+     '2ef1ad424d5f12a4d0ae06938da623819596cee7c0fb4616008f27583c29494d'},
+    {'PyTorch-2.9.1_skip-tests-requiring-MKLDNN.patch':
+     '03756a8069bad01018f422f41aa24c7c543519fd857db88a0c6de661976c8859'},
 ]
 
 osdependencies = [OS_PKG_IBVERBS_DEV]
 
 builddependencies = [
-    ('CMake', '3.29.3'),
-    ('hypothesis', '6.103.1'),
-    ('pybind11', '2.12.0'),
+    ('CMake', '3.31.8'),
+    ('hypothesis', '6.136.6'),
     # For tests
     ('parameterized', '0.9.0'),
     ('pytest-flakefinder', '1.1.0'),
-    ('pytest-rerunfailures', '15.0'),
-    ('pytest-shard', '0.1.2'),
-    ('pytest-subtests', '0.13.1'),
-    ('tlparse', '0.3.37'),
-    ('optree', '0.14.1'),
-    ('unittest-xml-reporting', '3.1.0'),
+    ('pytest-rerunfailures', '16.1'),
+    ('pytest-subtests', '0.15.0'),
+    ('tlparse', '0.4.3'),
+    ('optree', '0.18.0'),
+    ('unittest-xml-reporting', '3.2.0'),
 ]
 
 dependencies = [
-    ('CUDA', '12.6.0', '', SYSTEM),
+    ('CUDA', '12.9.1', '', SYSTEM),
     # PyTorch is very sensitive to the NCCL & cuDNN versions. (Maybe the same for cuSPARSELt)
     # Prefer those (listed per CUDA version) in
     # https://github.com/pytorch/pytorch/blob/main/.github/scripts/generate_binary_build_matrix.py
     # or https://github.com/pytorch/pytorch/blob/main/.ci/docker/common/install_cuda.sh
-    ('NCCL', '2.26.2', versionsuffix),
-    ('cuDNN', '9.5.1.17', versionsuffix, SYSTEM),
-    ('magma', '2.9.0', versionsuffix),
-    ('cuSPARSELt', '0.6.3.2', versionsuffix, SYSTEM),
-    ('nvidia-cutlass', '3.8.0.0', versionsuffix),
-    # Version from .ci/docker/triton_version.txt
-    ('Triton', '3.3.1', versionsuffix),
-    ('Ninja', '1.12.1'),  # Required for JIT compilation of C++ extensions
-    ('Python', '3.12.3'),
-    ('Python-bundle-PyPI', '2024.06'),
-    ('expecttest', '0.2.1'),
+    ('NCCL', '2.27.7', '-CUDA-%(cudaver)s'),
+    ('cuDNN', '9.15.0.57', '-CUDA-%(cudaver)s', SYSTEM),
+    ('magma', '2.9.0', '-CUDA-%(cudaver)s'),
+    ('cuSPARSELt', '0.8.0.4', '-CUDA-%(cudaver)s', SYSTEM),
+    ('Triton', '3.5.0', '-CUDA-%(cudaver)s'),
+    ('Ninja', '1.13.0'),
+    ('Python', '3.13.5'),
+    ('Python-bundle-PyPI', '2025.07'),
+    ('expecttest', '0.3.0'),
     ('GMP', '6.3.0'),
-    ('MPFR', '4.2.1'),
-    ('networkx', '3.4.2'),
-    ('numactl', '2.0.18'),
-    ('Pillow', '10.4.0'),
-    ('protobuf-python', '5.28.0'),
-    ('protobuf', '28.0'),
-    ('PuLP', '2.8.0'),
+    ('MPFR', '4.2.2'),
+    ('networkx', '3.5'),
+    ('numactl', '2.0.19'),
+    ('Pillow', '11.3.0'),
+    ('protobuf-python', '6.31.1'),
+    ('protobuf', '31.1'),
+    ('pybind11', '3.0.0'),
+    ('PuLP', '3.3.0'),
     ('PyYAML', '6.0.2'),
-    ('pyzstd', '0.16.2'),
-    ('SciPy-bundle', '2024.05'),
-    ('sympy', '1.13.3'),
-    ('Z3', '4.13.0',),
+    ('pyzstd', '0.19.0'),
+    ('SciPy-bundle', '2025.07'),
+    ('sympy', '1.14.0'),
+    ('Z3', '4.15.1'),
 ]
 
+prebuildopts = (f"""sed -i '1i set(PYTHON_SIX_SOURCE_DIR "%(builddir)s/six-{local_six_version}")' """
+                "cmake/Dependencies.cmake && ")
 buildcmd = '%(python)s setup.py build'  # Run the (long) build in the build step
 
 excluded_tests = {
@@ -282,16 +286,34 @@ excluded_tests = {
         'test_public_bindings',
         # 1 Failure and not important
         'dynamo/test_utils',
+        # Packaging test only, not important for us
+        'test_license',
+        # Occasional segfaults on CPU
+        'inductor/test_flex_attention',
+        'inductor/test_flex_decoding ',
     ]
 }
 
-local_test_opts = '--continue-through-error --pipe-logs --verbose %(excluded_tests)s'
-runtest = 'cd test && PYTEST_ADDOPTS=--full-trace PYTHONUNBUFFERED=1 %(python)s run_test.py ' + local_test_opts
+runtest = (
+    # Disable symbol resolution in stack traces that can cause hangs and slowdowns
+    ' TORCH_DISABLE_ADDR2LINE=1'
+    ' TORCHINDUCTOR_CUTLASS_DIR=%(start_dir)s/third_party/cutlass'
+    ' PYTEST_ADDOPTS=--full-trace'
+    ' PYTHONUNBUFFERED=1'
+    ' %(python)s test/run_test.py'
+    ' --continue-through-error --pipe-logs --verbose'
+    ' %(excluded_tests)s'
+)
+
+postinstallcmds = [
+    "mkdir %(installdir)s/extra",
+    "cp -r third_party/cutlass %(installdir)s/extra/",
+]
+
+modextrapaths = {'TORCHINDUCTOR_CUTLASS_DIR': 'extra/cutlass'}
 
-# ATTENTION: NVIDIA Volta not (fully) supported anymore.
-# Allow some more tests to fail, especially due to that
-max_failed_tests = 60
+tests = ['PyTorch-check-cpp-extension.py', 'PyTorch-check-cutlass.py']
 
-tests = ['PyTorch-check-cpp-extension.py']
+sanity_check_pip_list = False
 
 moduleclass = 'ai'
Diff against PyTorch-2.9.1-foss-2024a-CUDA-12.6.0.eb

easybuild/easyconfigs/p/PyTorch/PyTorch-2.9.1-foss-2024a-CUDA-12.6.0.eb

diff --git a/easybuild/easyconfigs/p/PyTorch/PyTorch-2.9.1-foss-2024a-CUDA-12.6.0.eb b/easybuild/easyconfigs/p/PyTorch/PyTorch-2.9.1-foss-2025b-CUDA-12.9.1.eb
index 3ef4f5d0f6..e38d4e84ff 100644
--- a/easybuild/easyconfigs/p/PyTorch/PyTorch-2.9.1-foss-2024a-CUDA-12.6.0.eb
+++ b/easybuild/easyconfigs/p/PyTorch/PyTorch-2.9.1-foss-2025b-CUDA-12.9.1.eb
@@ -6,7 +6,7 @@ homepage = 'https://pytorch.org/'
 description = """Tensors and Dynamic neural networks in Python with strong GPU acceleration.
 PyTorch is a deep learning framework that puts Python first."""
 
-toolchain = {'name': 'foss', 'version': '2024a'}
+toolchain = {'name': 'foss', 'version': '2025b'}
 
 local_six_version = '1.11.0'
 # This is specific to a (tagged) release.
@@ -26,7 +26,7 @@ sources = [
         # Avoid downloading this during the build, see third_party/NNPACK/cmake/DownloadSix.cmake for the version
         'filename': f'six-{local_six_version}.tar.gz',
         'source_urls': ['https://pypi.python.org/packages/source/s/six'],
-    }
+    },
 ]
 patches = [
     'PyTorch-1.12.1_add-hypothesis-suppression.patch',
@@ -57,32 +57,37 @@ patches = [
     'PyTorch-2.9.0_revert-pybind11-3-change.patch',
     'PyTorch-2.9.0_skip-test_benchmark_on_non_zero_device.patch',
     'PyTorch-2.9.0_skip-test_convolution1-on-H100.patch',
+    'PyTorch-2.9.1_skip-test_dtensor_op_db_nn_functional_multi_head_attention_forward_cpu_float32.patch',
     'PyTorch-2.9.0_skip-test_inductor_all_gather_into_tensor_coalesced.patch',
     'PyTorch-2.9.0_skip-test_original_aten_preserved_pad_mm.patch',
     'PyTorch-2.9.0_skip-test_override-without-CUDA.patch',
-    'PyTorch-2.9.0_skip-tests-requiring-CUDA-12.8.patch',
     'PyTorch-2.9.0_skip-test_unbacked_reduction.patch',
+    'PyTorch-2.9.0_skip-tests-requiring-CUDA-12.8.patch',
     'PyTorch-2.9.0_skip-unexpected-success-in-test_fake_export.patch',
-    'PyTorch-2.9.0_update-exptected-output-for-z3-4.13.0.patch',
     'PyTorch-2.9.1_avoid-multiprocess-tests-hanging-forever.patch',
     'PyTorch-2.9.1_avoid-using-wrong-libomp.patch',
     'PyTorch-2.9.1_check-device-avail-test_schedule.patch',
     'PyTorch-2.9.1_disable-slow-tests.patch',
     'PyTorch-2.9.1_dont-print-test-items.patch',
+    'PyTorch-2.9.1_fix-DDPCommHookType-python-3.13.patch',
     'PyTorch-2.9.1_fix-hypothesis-deadline.patch',
     'PyTorch-2.9.1_fix-iteration-in-fligh-reporter.patch',
-    'PyTorch-2.9.1_fix-test_dist2-decorators.patch',
+    'PyTorch-2.9.1_fix-pickle-error-on-Python-3.13.patch',
     'PyTorch-2.9.1_fix-TestExportOpInfoCPU-with-single-GPU.patch',
+    'PyTorch-2.9.1_fix-test_dist2-decorators.patch',
+    'PyTorch-2.9.1_fix-test_recursion_in_except_handler.patch',
     'PyTorch-2.9.1_GCC14-ARM-workaround.patch',
     'PyTorch-2.9.1_ignore-warning-incompatible-pointer-types.patch',
     'PyTorch-2.9.1_normalize_tree_output.patch',
     'PyTorch-2.9.1_set-test-timeout.patch',
+    'PyTorch-2.9.1_skip-bool-bessel-tests.patch',
     'PyTorch-2.9.1_skip-cutlass-addmm-test.patch',
-    'PyTorch-2.9.1_skip-flex-attention-test_block_mask_non_divisible.patch',
     'PyTorch-2.9.1_skip-flex-attention-tests-on-unsupported-cpus.patch',
-    'PyTorch-2.9.1_skip-RingFlexAttentionTest.patch',
-    'PyTorch-2.9.1_skip-test_dtensor_op_db_nn_functional_multi_head_attention_forward_cpu_float32.patch',
+    'PyTorch-2.9.1_skip-flex-attention-test_block_mask_non_divisible.patch',
+    'PyTorch-2.9.1_skip-test_checkpoint_save_failure_continues_serving.patch',
+    'PyTorch-2.9.1_skip-test_norm_matrix_degenerate_shapes.patch',
     'PyTorch-2.9.1_skip-tests-requiring-SM90.patch',
+    'PyTorch-2.9.1_skip-RingFlexAttentionTest.patch',
     'PyTorch-2.9.1_increase-tolerance-TestDecomp-matmul.patch',
     'PyTorch-2.9.1_skip-cpu_repro-tests-failing-on-ARM.patch',
     'PyTorch-2.9.1_skip-svd-pca-lowrank-tests-on-cpu.patch',
@@ -139,20 +144,20 @@ checksums = [
      '85e236431d1a5da3fb7fccc2554640898c29f5fab46a41d15b3ab61dd1f924fc'},
     {'PyTorch-2.9.0_skip-test_convolution1-on-H100.patch':
      '704750c7cc08b58779907d608cd4b7505043e394fb27530b16d72a0dc27c277e'},
+    {'PyTorch-2.9.1_skip-test_dtensor_op_db_nn_functional_multi_head_attention_forward_cpu_float32.patch':
+     'e57486cc42f3dbcae29753168febc251d070a283229e2d76ccbdf19fee53f06e'},
     {'PyTorch-2.9.0_skip-test_inductor_all_gather_into_tensor_coalesced.patch':
      '644153d4c1d8267c0631df2902a6dfe8ec2a197f3374f2a2f5654e6bd0edc05e'},
     {'PyTorch-2.9.0_skip-test_original_aten_preserved_pad_mm.patch':
      'ac9e05d296cd5ff938a44662cd022efcc8133c744ca82b045c6a15bc64f67cf4'},
     {'PyTorch-2.9.0_skip-test_override-without-CUDA.patch':
      '967512d1487bf1ad06982cc5b976c0b38ba062c3f3473cb4542c4b9ac0740662'},
-    {'PyTorch-2.9.0_skip-tests-requiring-CUDA-12.8.patch':
-     '6d79aff5291627b86d8fea025bf2379e4065c7d9cbef5cf83452c35922848728'},
     {'PyTorch-2.9.0_skip-test_unbacked_reduction.patch':
      'b51dd5d7c9cfeed946cbc5c7fc22f2e78e1fa52dda55569b957c20ca4ed01fe8'},
+    {'PyTorch-2.9.0_skip-tests-requiring-CUDA-12.8.patch':
+     '6d79aff5291627b86d8fea025bf2379e4065c7d9cbef5cf83452c35922848728'},
     {'PyTorch-2.9.0_skip-unexpected-success-in-test_fake_export.patch':
      '2e73f71ea0f09e613cc4a108893e7948b6daf239e3fe42fd2d3ae5d43c3cf9de'},
-    {'PyTorch-2.9.0_update-exptected-output-for-z3-4.13.0.patch':
-     '5c68e0de73212ed266879f4528a6041ef7ab2f1ac83c6cf7142c4baa78e7664c'},
     {'PyTorch-2.9.1_avoid-multiprocess-tests-hanging-forever.patch':
      '86ce380e69b3b20e010d817889cb1b825b05b4054a045b00f2ac12161b77d7e4'},
     {'PyTorch-2.9.1_avoid-using-wrong-libomp.patch':
@@ -161,30 +166,40 @@ checksums = [
      '64c28d38ce69147565509add36d310473ce46f14a0a876d38b5049cb7fce9817'},
     {'PyTorch-2.9.1_disable-slow-tests.patch': '6b365a3531b0ac5446b5f0e8ab924b5e5742cd0331e6d9ec979118a3ef0ffc09'},
     {'PyTorch-2.9.1_dont-print-test-items.patch': '2b524cf3d557c0672feefc3a7165e5555e549b0720647a84d546f769cea0be07'},
-    {'PyTorch-2.9.1_fix-hypothesis-deadline.patch': 'f7a130669eee9924a303df9e2bd5743ff023a7d994b7a3e43c86dcccf0206c49'},
+    {'PyTorch-2.9.1_fix-DDPCommHookType-python-3.13.patch':
+     'd7bafe8340bba9dd909475fc62b739b0ce3f95d3409479ef8c5929351dd2a05d'},
+    {'PyTorch-2.9.1_fix-hypothesis-deadline.patch':
+     'f7a130669eee9924a303df9e2bd5743ff023a7d994b7a3e43c86dcccf0206c49'},
     {'PyTorch-2.9.1_fix-iteration-in-fligh-reporter.patch':
      'ab408275ec66e836112a50054acc4e789ef38196efeb6137c6061d60d9ac9ead'},
-    {'PyTorch-2.9.1_fix-test_dist2-decorators.patch':
-     'bf4ed805f00775ed33351de7bce40ebf4eac16aff6c61d2e91790982bc43d73b'},
+    {'PyTorch-2.9.1_fix-pickle-error-on-Python-3.13.patch':
+     '88807b5564485968de3be6411d33c257c5ce59f5d3db23c7aeba884458102d57'},
     {'PyTorch-2.9.1_fix-TestExportOpInfoCPU-with-single-GPU.patch':
      'bdddf5a9ba47d57ec96f4bbefc3b85c4904e44de93dc5c7a65bc03e343035ae9'},
+    {'PyTorch-2.9.1_fix-test_dist2-decorators.patch':
+     'bf4ed805f00775ed33351de7bce40ebf4eac16aff6c61d2e91790982bc43d73b'},
+    {'PyTorch-2.9.1_fix-test_recursion_in_except_handler.patch':
+     'e7a64dbdc202151c5bff6aac86d77b0f6e7c52dc3117e3bfe9b57ec1371f87ad'},
     {'PyTorch-2.9.1_GCC14-ARM-workaround.patch': 'ea8a8662e20fae2fb3a74c7f8bf390aba80a598ab37f9131c720d25ebb14965d'},
     {'PyTorch-2.9.1_ignore-warning-incompatible-pointer-types.patch':
      'c4dad43a5d76e292bb0cada56ea05e8cbd522e3e83749cf3b2c15cd1e4ff6d7b'},
     {'PyTorch-2.9.1_normalize_tree_output.patch': '7d5994580339b73c28de595d9e5a0448db97b7d284f17efd18909e4613d170df'},
     {'PyTorch-2.9.1_set-test-timeout.patch': '15fa1149c250b1333b0bc491f659aaf89d5d6eaf6df5ebc81eea545478c1239c'},
+    {'PyTorch-2.9.1_skip-bool-bessel-tests.patch': '9c07cddaf4c1b17fe9a54874f10b8edbfb85785c40ac1e3aea11c7f4b5abca69'},
     {'PyTorch-2.9.1_skip-cutlass-addmm-test.patch':
      '1f81a8a9eea8eda51fc93dff84cd994772febf4fd05d77efbf21b8440dadfd4e'},
-    {'PyTorch-2.9.1_skip-flex-attention-test_block_mask_non_divisible.patch':
-     'd8489c192da549083569e09e5f94d2a83c9e41e111b1322f86512a9c5a58c0d9'},
     {'PyTorch-2.9.1_skip-flex-attention-tests-on-unsupported-cpus.patch':
      'e544f765beac7bdb3fc0ada98a3f92fd7e511ed8874de085aa2f213cca769d40'},
-    {'PyTorch-2.9.1_skip-RingFlexAttentionTest.patch':
-     '3cf0b11136fb18c45072687eafd3024d91b504d231a4fa40e04bc62d8d6019c7'},
-    {'PyTorch-2.9.1_skip-test_dtensor_op_db_nn_functional_multi_head_attention_forward_cpu_float32.patch':
-     'e57486cc42f3dbcae29753168febc251d070a283229e2d76ccbdf19fee53f06e'},
+    {'PyTorch-2.9.1_skip-flex-attention-test_block_mask_non_divisible.patch':
+     'd8489c192da549083569e09e5f94d2a83c9e41e111b1322f86512a9c5a58c0d9'},
+    {'PyTorch-2.9.1_skip-test_checkpoint_save_failure_continues_serving.patch':
+     'fa22d7ed5bf20afa4798c8af3ec732b1a3f530ecc4be5c223b3796e839b0b812'},
+    {'PyTorch-2.9.1_skip-test_norm_matrix_degenerate_shapes.patch':
+     'd6082e62696a38dbfbc87c228f7ccb54dba4cfc615ce158f1f3bf77e6e30ff4f'},
     {'PyTorch-2.9.1_skip-tests-requiring-SM90.patch':
      '7db02152db2ae70c0fd4c4602fe381e26a74b8e4f7b16b1a3554b2353d761b10'},
+    {'PyTorch-2.9.1_skip-RingFlexAttentionTest.patch':
+     '3cf0b11136fb18c45072687eafd3024d91b504d231a4fa40e04bc62d8d6019c7'},
     {'PyTorch-2.9.1_increase-tolerance-TestDecomp-matmul.patch':
      'dd82203ce3b6262255aba6b59fb3b547c4c17875d5711f6d3d489aa8f0f59f32'},
     {'PyTorch-2.9.1_skip-cpu_repro-tests-failing-on-ARM.patch':
@@ -200,49 +215,47 @@ checksums = [
 osdependencies = [OS_PKG_IBVERBS_DEV]
 
 builddependencies = [
-    ('CMake', '3.29.3'),
-    ('hypothesis', '6.103.1'),
-    ('setuptools', '80.9.0'),
+    ('CMake', '3.31.8'),
+    ('hypothesis', '6.136.6'),
     # For tests
     ('parameterized', '0.9.0'),
     ('pytest-flakefinder', '1.1.0'),
-    ('pytest-rerunfailures', '15.0'),
-    ('pytest-subtests', '0.13.1'),
-    ('tlparse', '0.4.0'),
-    ('optree', '0.14.1'),
-    ('unittest-xml-reporting', '3.1.0'),
+    ('pytest-rerunfailures', '16.1'),
+    ('pytest-subtests', '0.15.0'),
+    ('tlparse', '0.4.3'),
+    ('optree', '0.18.0'),
+    ('unittest-xml-reporting', '3.2.0'),
 ]
 
 dependencies = [
-    ('CUDA', '12.6.0', '', SYSTEM),
+    ('CUDA', '12.9.1', '', SYSTEM),
     # PyTorch is very sensitive to the NCCL & cuDNN versions. (Maybe the same for cuSPARSELt)
     # Prefer those (listed per CUDA version) in
     # https://github.com/pytorch/pytorch/blob/main/.github/scripts/generate_binary_build_matrix.py
     # or https://github.com/pytorch/pytorch/blob/main/.ci/docker/common/install_cuda.sh
-    ('NCCL', '2.27.5', versionsuffix),
-    ('cuDNN', '9.10.2.21', versionsuffix, SYSTEM),
-    ('magma', '2.9.0', versionsuffix),
-    ('cuSPARSELt', '0.6.3.2', versionsuffix, SYSTEM),
-    # Version from .ci/docker/triton_version.txt
-    ('Triton', '3.5.0', versionsuffix),
-    ('Ninja', '1.12.1'),  # Required for JIT compilation of C++ extensions
-    ('Python', '3.12.3'),
-    ('Python-bundle-PyPI', '2024.06'),
-    ('expecttest', '0.2.1'),
+    ('NCCL', '2.27.7', '-CUDA-%(cudaver)s'),
+    ('cuDNN', '9.15.0.57', '-CUDA-%(cudaver)s', SYSTEM),
+    ('magma', '2.9.0', '-CUDA-%(cudaver)s'),
+    ('cuSPARSELt', '0.8.0.4', '-CUDA-%(cudaver)s', SYSTEM),
+    ('Triton', '3.5.0', '-CUDA-%(cudaver)s'),
+    ('Ninja', '1.13.0'),
+    ('Python', '3.13.5'),
+    ('Python-bundle-PyPI', '2025.07'),
+    ('expecttest', '0.3.0'),
     ('GMP', '6.3.0'),
-    ('MPFR', '4.2.1'),
-    ('networkx', '3.4.2'),
-    ('numactl', '2.0.18'),
-    ('Pillow', '10.4.0'),
-    ('protobuf-python', '5.28.0'),
-    ('protobuf', '28.0'),
-    ('pybind11', '2.12.0'),
-    ('PuLP', '2.8.0'),
+    ('MPFR', '4.2.2'),
+    ('networkx', '3.5'),
+    ('numactl', '2.0.19'),
+    ('Pillow', '11.3.0'),
+    ('protobuf-python', '6.31.1'),
+    ('protobuf', '31.1'),
+    ('pybind11', '3.0.0'),
+    ('PuLP', '3.3.0'),
     ('PyYAML', '6.0.2'),
-    ('pyzstd', '0.16.2'),
-    ('SciPy-bundle', '2024.05'),
-    ('sympy', '1.13.3'),
-    ('Z3', '4.13.0',),
+    ('pyzstd', '0.19.0'),
+    ('SciPy-bundle', '2025.07'),
+    ('sympy', '1.14.0'),
+    ('Z3', '4.15.1'),
 ]
 
 prebuildopts = (f"""sed -i '1i set(PYTHON_SIX_SOURCE_DIR "%(builddir)s/six-{local_six_version}")' """
@@ -301,4 +314,6 @@ modextrapaths = {'TORCHINDUCTOR_CUTLASS_DIR': 'extra/cutlass'}
 
 tests = ['PyTorch-check-cpp-extension.py', 'PyTorch-check-cutlass.py']
 
+sanity_check_pip_list = False
+
 moduleclass = 'ai'

@Thyre

This comment was marked as outdated.

@Thyre

This comment was marked as resolved.

@Flamefire
Copy link
Copy Markdown
Contributor Author

Test report by @Thyre FAILED Build succeeded for 3 out of 4 (total: 55 secs) (4 easyconfigs in total) jrc0900.jureca - Linux Rocky Linux 9.6, AArch64, ARM UNKNOWN (neoverse_v2), 1 x NVIDIA NVIDIA GH200 480GB, 580.95.05, Python 3.9.21 See https://gist.github.com/Thyre/576f0dbeceb975733d860d97f16ca3fc for a full test report.

== 2025-12-19 10:19:27,773 build_log.py:233 ERROR EasyBuild encountered an error: Nothing found to replace 'if IS_CI:\n\s+# Add the option to generate XML test report.*' in test/run_test.py (at easybuild/tools/filetools.py:1861 in apply_regex_substitutions)

Are you using the latest easyblock? It is missing this commit from easybuilders/easybuild-easyblocks#3803

@Thyre

This comment was marked as outdated.

@Flamefire
Copy link
Copy Markdown
Contributor Author

2025b is using GCC 14 that has new warnings. See pytorch/pytorch#166873

Patch added. Seems to only affect ARM

@Thyre

This comment was marked as outdated.

@Flamefire
Copy link
Copy Markdown
Contributor Author

Oh, it is a C file. Updated the patch to also add it to C-flags

@Thyre

This comment was marked as outdated.

@Flamefire
Copy link
Copy Markdown
Contributor Author

Looks like I need to set those values earlier. Can you try again?

@Thyre
Copy link
Copy Markdown
Collaborator

Thyre commented Dec 19, 2025

Actual failure was an internal GCC compiler error:

In file included from /dev/shm/reuter1/easybuild/build/PyTorch/2.9.1/foss-2025b-CUDA-12.9.1/pytorch-v2.9.1/build/aten/src/ATen/native/cpu/Unfold2d.cpp.SVE256.cpp:1:
/dev/shm/reuter1/easybuild/build/PyTorch/2.9.1/foss-2025b-CUDA-12.9.1/pytorch-v2.9.1/aten/src/ATen/native/cpu/Unfold2d.cpp: In function ‘void at::native::{anonymous}::unfolded2d_acc_kernel(c10::ScalarType, void*, void*, int64_t, int64_t, int64_t, int64_t, int64_t, int64_t, int64_t, int64_t, int64_t, int64_t, int64_t, bool)’:
/dev/shm/reuter1/easybuild/build/PyTorch/2.9.1/foss-2025b-CUDA-12.9.1/pytorch-v2.9.1/aten/src/ATen/native/cpu/Unfold2d.cpp:225:1: error: unrecognizable insn:
  225 | }
      | ^
(insn 1375 1374 1376 99 (set (reg:VNx16BI 3253)
        (unspec:VNx16BI [
                (reg:VNx16BI 3250)
                (reg:VNx8BI 3252)
                (const_vector:VNx4BI [
                        (const_int 0 [0]) repeated x8
                    ])
            ] UNSPEC_TRN1_CONV)) "/dev/shm/reuter1/easybuild/build/PyTorch/2.9.1/foss-2025b-CUDA-12.9.1/pytorch-v2.9.1/torch/headeronly/util/bit_cast.h":40:14 -1
     (nil))
during RTL pass: vregs
/dev/shm/reuter1/easybuild/build/PyTorch/2.9.1/foss-2025b-CUDA-12.9.1/pytorch-v2.9.1/aten/src/ATen/native/cpu/Unfold2d.cpp:225:1: internal compiler error: in extract_insn, at recog.cc:2812
0x7d30df _fatal_insn(char const*, rtx_def const*, char const*, int, char const*)
	../../gcc/rtl-error.cc:108
0x7d3113 _fatal_insn_not_found(rtx_def const*, char const*, int, char const*)
	../../gcc/rtl-error.cc:116
0xec1d17 extract_insn(rtx_insn*)
	../../gcc/recog.cc:2812
0xc2a28b instantiate_virtual_regs_in_insn
	../../gcc/function.cc:1612
0xc2a28b instantiate_virtual_regs
	../../gcc/function.cc:1995
0xc2a28b execute
	../../gcc/function.cc:2042
Please submit a full bug report, with preprocessed source (by using -freport-bug).
Please include the complete backtrace with any bug report.
See <https://gcc.gnu.org/bugs/> for instructions.

Test report by @Thyre
FAILED
Build succeeded for 3 out of 4 (total: 17 mins 7 secs) (4 easyconfigs in total)
jrc0900.jureca - Linux Rocky Linux 9.6, AArch64, ARM UNKNOWN (neoverse_v2), 1 x NVIDIA NVIDIA GH200 480GB, 580.95.05, Python 3.9.21
See https://gist.github.com/Thyre/bdc1ee06d4f8b430f52f9c220b66e11f for a full test report.

@Thyre
Copy link
Copy Markdown
Collaborator

Thyre commented Dec 19, 2025

@Flamefire
Copy link
Copy Markdown
Contributor Author

Flamefire commented Dec 19, 2025

Failure may be caused by this: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=121027

There was a PR which should have worked around this, but seemingly the fix doesn't work? See also:

* https://github.com/pytorch/pytorch/blob/f026b098e4319413db7d3fc1dbcb39dda69fcf0c/aten/src/ATen/native/cpu/Unfold2d.cpp#L172

* [Build error: unrecognizable insn with using gcc-14 on aarch64 pytorch/pytorch#157842](https://github.com/pytorch/pytorch/issues/157842)

That is not included in this (or any) release yet. I'll add it to the patch list

Maybe we need to patch GCCcore/14.3.0 with this change? https://gcc.gnu.org/bugzilla/show_bug.cgi?id=121027#c9

Would be an option, not sure if it is worth it: This EC is included since EB 5.1.0, although we did that in the past

@Flamefire Flamefire changed the title {tools}[GCCcore/14.3.0] parameterized v0.9.0, pytest-subtests v0.15.0, PyTorch v2.9.1, ... w/ CUDA 12.9.1 {tools}[GCCcore/14.3.0] PyTorch v2.9.1, parameterized v0.9.0, pytest-subtests v0.15.0, ... w/ CUDA 12.9.1 Dec 19, 2025
@boegel boegel added this to the next release (5.2.1?) milestone Dec 31, 2025
@github-actions github-actions Bot added the 2024a issues & PRs related to 2024a common toolchains label Jan 15, 2026
@verdurin
Copy link
Copy Markdown
Member

verdurin commented Mar 5, 2026

Worth noting here that I've been trying this on GH200 nodes, and the testing was taking over a week when the node died for other reasons.

@Flamefire
Copy link
Copy Markdown
Contributor Author

Test report by @Flamefire
SUCCESS
Build succeeded for 1 out of 1 (total: 23 hours 44 mins 43 secs) (1 easyconfigs in total)
c51 - Linux Rocky Linux 9.6, x86_64, AMD EPYC 9334 32-Core Processor (zen4), 4 x NVIDIA NVIDIA H100, 580.65.06, Python 3.9.21
See https://gist.github.com/Flamefire/5f82b6abcf65a7e9d08b948be2baad10 for a full test report.

@Flamefire
Copy link
Copy Markdown
Contributor Author

Test report by @Flamefire
FAILED
Build succeeded for 0 out of 1 (total: 44 hours 28 mins 55 secs) (1 easyconfigs in total)
i8012 - Linux Rocky Linux 9.6, x86_64, AMD EPYC 7352 24-Core Processor (zen2), 8 x NVIDIA NVIDIA A100-SXM4-40GB, 580.65.06, Python 3.9.21
See https://gist.github.com/Flamefire/f6896e1e8697116ae98e2cd9f64f77f1 for a full test report.

@Flamefire Flamefire force-pushed the 20251218180340_new_pr_parameterized090 branch from f32c592 to c30732f Compare March 11, 2026 11:56
@Flamefire
Copy link
Copy Markdown
Contributor Author

Flamefire commented Mar 11, 2026

Test report by @Flamefire
FAILED
Build succeeded for 0 out of 1 (total: 44 hours 28 mins 55 secs) (1 easyconfigs in total)
i8012 - Linux Rocky Linux 9.6, x86_64, AMD EPYC 7352 24-Core Processor (zen2), 8 x NVIDIA NVIDIA A100-SXM4-40GB, 580.65.06, Python 3.9.21
See https://gist.github.com/Flamefire/f6896e1e8697116ae98e2cd9f64f77f1 for a full test report.

There were 17 test failures.

distributed/algorithms/ddp_comm_hooks/test_ddp_hooks (5 failed, 1 passed, 0 skipped, 0 errors)

Fixed by PyTorch-2.9.1_fix-DDPCommHookType-python-3.13.patch

distributed/checkpoint/e2e/test_e2e_save_and_load (1 failed, 18 passed, 0 skipped, 0 errors)
distributed/checkpoint/test_checkpoint (2 failed, 6 passed, 0 skipped, 0 errors)
distributed/checkpoint/test_fsspec (1 failed, 2 passed, 0 skipped, 0 errors)

Issue with new Python version ("cannot pickle code object") which I'd just let fail

distributed/pipelining/test_stage (1 failed, 1 passed, 6 skipped, 0 errors)

Happens on nodes with > 4 GPUs: pytorch/pytorch#162746

dynamo/test_error_messages (1 failed, 40 passed, 0 skipped, 0 errors)

pybind11 ABI changes while test checks for exact strings

functorch/test_ops (2 failed, 7489 passed, 2731 skipped, 0 errors)

Known failure but PyTorch-2.1.0_increase-tolerance-functorch-test_vmapvjpvjp.patch can't be easily updated

inductor/test_aot_inductor (4 failed, 418 passed, 445 skipped, 0 errors)

IMPORTANT: cudaErrorInvalidDeviceFunction depending on the order of --cuda-compute-capabilities, fixed by easybuilders/easybuild-easyblocks#4092

Could not count failed tests for the following test suites/files:
inductor/test_flex_attention (Undetected or did not run properly)

I tested this many times interactively and can't reproduce the segfaults nor find any hint for a cause, so I just skipped that.

So 9 out of 17 and the uncounted suite fixed, the remaining 8 should be ok to keep.

@Flamefire
Copy link
Copy Markdown
Contributor Author

Issue with new Python version ("cannot pickle code object") which I'd just let fail

Fixed by using patch from upstream now

@boegel boegel changed the title {tools}[GCCcore/14.3.0] PyTorch v2.9.1, parameterized v0.9.0, pytest-subtests v0.15.0, ... w/ CUDA 12.9.1 {tools}[foss/2025b] PyTorch v2.9.1, parameterized v0.9.0, pytest-subtests v0.15.0, ... w/ CUDA 12.9.1 Mar 25, 2026
@boegel
Copy link
Copy Markdown
Member

boegel commented Mar 27, 2026

Test report by @boegel
SUCCESS
Build succeeded for 4 out of 4 (total: 37 hours 7 mins 12 secs) (1 easyconfigs in total)
node3907.accelgor.os - Linux RHEL 9.6, x86_64, AMD EPYC 7413 24-Core Processor (zen3), 1 x NVIDIA NVIDIA A100-SXM4-80GB, 590.48.01, Python 3.9.21
See https://gist.github.com/boegel/060a1ce806826b1d04f9b4e7a639fe9f for a full test report.

@Flamefire
Copy link
Copy Markdown
Contributor Author

After the benchmarks it seems like we should include ACL and MKL in our builds. Shall we merge this first and make new PRs for adding them or add them here (and in other open PyTorch PRs) before merging?

@boegel
Copy link
Copy Markdown
Member

boegel commented Mar 30, 2026

After the benchmarks it seems like we should include ACL and MKL in our builds. Shall we merge this first and make new PRs for adding them or add them here (and in other open PyTorch PRs) before merging?

I'd say we can do that in a separate PR, to make the change/fix stand out?

@boegel
Copy link
Copy Markdown
Member

boegel commented Mar 30, 2026

@Flamefire please sync with develop now #24365 is merged...

@boegel
Copy link
Copy Markdown
Member

boegel commented Mar 30, 2026

After the benchmarks it seems like we should include ACL and MKL in our builds. Shall we merge this first and make new PRs for adding them or add them here (and in other open PyTorch PRs) before merging?

I'd say we can do that in a separate PR, to make the change/fix stand out?

Also, it's not clear to me yet if it's only imkl or also oneDNN (see also https://github.com/uxlfoundation/oneDNN ?)

Just for context:

@Flamefire
Copy link
Copy Markdown
Contributor Author

I'd say we can do that in a separate PR, to make the change/fix stand out?

It was a change to yet unmerged ECs, hence the question. But of course can do in separate PRs after having those merged. Makes the changeset smaller.

Also, it's not clear to me yet if it's only imkl or also oneDNN (see also uxlfoundation/oneDNN ?)

oneDNN is included in PyTorch as a submodule and already used, but it is (sometimes) named MKL-DNN/MKLDNN which is the previous name: https://www.intel.com/content/www/us/en/docs/onednn/developer-guide-reference/2024-1/transitioning-from-intel-mkl-dnn-to-onednn.html

In short, the migration can be as simple as just replacing all MKLDNN/mkldnn substrings with DNNL/dnnl.

https://github.com/intel/mkl-dnn forwards to https://github.com/uxlfoundation/oneDNN

@boegel
Copy link
Copy Markdown
Member

boegel commented Mar 31, 2026

Test report by @boegel
SUCCESS
Build succeeded for 1 out of 1 (total: 33 hours 22 mins 13 secs) (1 easyconfigs in total)
node4306.litleo.os - Linux RHEL 9.6, x86_64, AMD EPYC 9454P 48-Core Processor (zen4), 1 x NVIDIA NVIDIA H100 NVL, 580.95.05, Python 3.9.21
See https://gist.github.com/boegel/40208332798ff165baf4ad663c41f19f for a full test report.

@boegel
Copy link
Copy Markdown
Member

boegel commented Apr 8, 2026

@boegelbot please test @ jsc-zen3-a100
CORE_CNT=16

@boegelbot
Copy link
Copy Markdown
Collaborator

@boegel: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=24926 EB_ARGS= EB_CONTAINER= EB_REPO=easybuild-easyconfigs EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_24926 --ntasks="16" --partition=jsczen3g --gres=gpu:1 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 10183

Test results coming soon (I hope)...

Details

- notification for comment with ID 4208724136 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link
Copy Markdown
Collaborator

Test report by @boegelbot
SUCCESS
Build succeeded for 1 out of 1 (total: 43 hours 37 mins 2 secs) (1 easyconfigs in total)
jsczen3g1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.7, x86_64, AMD EPYC-Milan Processor (zen3), 1 x NVIDIA NVIDIA A100 80GB PCIe, 590.48.01, Python 3.9.25
See https://gist.github.com/boegelbot/978f1935f454debbe99adab94eff8ab4 for a full test report.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

2025b issues & PRs related to 2025b common toolchains update

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants